Skip to content

fix(#159): retry fullsend post-review on transient API failures#161

Open
fullsend-ai-coder[bot] wants to merge 3 commits into
mainfrom
agent/159-retry-post-review
Open

fix(#159): retry fullsend post-review on transient API failures#161
fullsend-ai-coder[bot] wants to merge 3 commits into
mainfrom
agent/159-retry-post-review

Conversation

@fullsend-ai-coder

Copy link
Copy Markdown
Contributor

Wrap the fullsend post-review call in a retry loop (max 3 attempts with 5s/15s backoff) to handle transient GitHub API errors such as 422 Unprocessable Entity during PR transitional states.

Previously, a single non-zero exit (other than code 10 for stale-head) caused post-review.sh to exit immediately. This meant a valid review comment could be posted while the formal review API call failed, leaving the PR without an outcome label and requiring human diagnosis.

Changes:

  • scripts/post-review.sh: add retry loop around fullsend post-review.
    Exit code 10 (stale-head) bypasses retry and uses existing handler.
    On exhausted retries, a degraded-mode fallback applies the outcome
    label directly so the PR is not left in limbo. The script still
    exits 1 to signal the formal review was not posted.
  • scripts/post-review-test.sh: add 7 integration tests covering
    retry-then-succeed, retries-exhausted with fallback for approve/
    comment/request-changes actions, stale-head bypass, and retry
    log message format.

POST_REVIEW_RETRY_DELAY env var overrides backoff for all retries (set to 0 in tests to skip sleep).


Closes #159

Post-script verification

  • Branch is not main/master (agent/159-retry-post-review)
  • Secret scan passed (gitleaks — 51561b23f669baf5e6ad027c5de39b0fdb199ab9..HEAD)
  • Pre-commit hooks passed (authoritative run on runner)
  • Tests ran inside sandbox

@fullsend-ai-coder
fullsend-ai-coder Bot requested a review from a team as a code owner July 13, 2026 20:04
@rh-hemartin

Copy link
Copy Markdown
Member

/fs-fix rebase and resolve the conficts.

@rh-hemartin rh-hemartin self-assigned this Jul 20, 2026
@fullsend-ai-coder

fullsend-ai-coder Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 11:31 AM UTC · Completed 11:37 AM UTC
Commit: 7d0bfa9 · View workflow run →

Wrap the fullsend post-review call in a retry loop (max 3 attempts
with 5s/15s backoff) to handle transient GitHub API errors such as
422 Unprocessable Entity during PR transitional states.

Previously, a single non-zero exit (other than code 10 for stale-head)
caused post-review.sh to exit immediately. This meant a valid review
comment could be posted while the formal review API call failed,
leaving the PR without an outcome label and requiring human diagnosis.

Changes:
- scripts/post-review.sh: add retry loop around fullsend post-review.
  Exit code 10 (stale-head) bypasses retry and uses existing handler.
  On exhausted retries, a degraded-mode fallback applies the outcome
  label directly so the PR is not left in limbo. The script still
  exits 1 to signal the formal review was not posted.
- scripts/post-review-test.sh: add 7 integration tests covering
  retry-then-succeed, retries-exhausted with fallback for approve/
  comment/request-changes actions, stale-head bypass, and retry
  log message format.

POST_REVIEW_RETRY_DELAY env var overrides backoff for all retries
(set to 0 in tests to skip sleep).

Closes #159
@fullsend-ai-coder
fullsend-ai-coder Bot force-pushed the agent/159-retry-post-review branch from 3327064 to 313ef13 Compare July 20, 2026 11:37
@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 1 (human-triggered)

Rebased onto origin/main, resolved conflict in post-review-test.sh by merging both draft PR tests and retry tests, and updated degraded-mode fallback to include PR_IS_DRAFT check for consistency with main's outcome-label logic. All 58 tests pass.

Fixed (1):

  1. rebase and resolve conflicts: Rebased branch onto origin/main (57 commits behind). Resolved merge conflict in scripts/post-review-test.sh by keeping both the draft PR / no-op label cycle tests (from main) and the retry logic tests (from the branch). Updated the degraded-mode fallback in post-review.sh to account for PR_IS_DRAFT, keeping it in sync with the main outcome-label block.

Tests: passed

Decision points
  • Updated degraded-mode fallback label logic to include PR_IS_DRAFT check (alternatives: Leave fallback without draft awareness, Add draft check to match main outcome-label block; rationale: The fallback comment says 'Label logic mirrors the outcome-label block below — keep in sync'. Main added PR_IS_DRAFT awareness to the outcome-label block; the fallback must match to prevent draft PRs from receiving ready-for-merge via the degraded path.)

Updated by fullsend fix agent

@rh-hemartin

Copy link
Copy Markdown
Member

/fs-review

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 22, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 6:31 AM UTC · Completed 6:48 AM UTC
Commit: 7ada4e0 · View workflow run →

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 22, 2026

Copy link
Copy Markdown

Review

Findings

Medium

Low

  • [scope-creep] scripts/post-review.sh — The fallback applies labels for all actions (approve, comment, reject), but issue Post-review.sh should retry on transient GitHub API errors when submitting formal PR reviews #159 only specifically authorized fallback for the approve→ready-for-merge case. The extension to other actions is a reasonable engineering decision but exceeds the issue's explicit scope.

  • [missing-requirement] scripts/post-review.sh — Issue Post-review.sh should retry on transient GitHub API errors when submitting formal PR reviews #159 requirement (4) asks for HTTP status code and response body in error messages, but the retry loop only logs the exit code. This is a CLI limitation acknowledged in the code comments (lines 1396–1399), not a script deficiency.

  • [code-duplication] scripts/post-review.sh — The degraded-mode fallback label logic (lines 1406–1414) duplicates the outcome-label determination from the normal path (lines 1483–1490). A "keep in sync" comment exists but does not eliminate maintenance risk. The test file already has a determine_outcome_label() function that could be promoted to reduce duplication.

Previous run

Review

Findings

Medium

  • [correctness-bug] scripts/post-review.sh — The degraded-mode fallback applies outcome labels after retry exhaustion without verifying whether fullsend post-review successfully posted the review comment to GitHub. The code comment acknowledges this ambiguity ("the review comment may have been posted"), but if all retries failed before posting anything, applying ready-for-merge would misrepresent the PR state. Mitigated by exit 1 ensuring CI reports failure regardless.
    Remediation: If fullsend post-review can emit a distinct exit code for "comment posted, formal review failed," use that to gate the fallback. Otherwise, document the assumption explicitly in the comment block.

  • [logic-error] scripts/post-review.sh — The degraded-mode fallback for the reject action applies the rejected label but does not close the PR, whereas the normal outcome-label path calls gh pr close with a comment before applying the label. The comment says "Label logic mirrors the outcome-label block below — keep in sync" but the close action is missing. In a degraded-mode reject scenario, the PR would be left open with a rejected label.
    Remediation: Either add gh pr close to the fallback's reject branch to mirror the normal path, or update the comment to explicitly note that closing is intentionally omitted in degraded mode as a conservative measure.

  • [protected-path] scripts/post-review.sh, scripts/post-review-test.sh — This PR modifies files under scripts/, which is a protected path. The PR links to issue Post-review.sh should retry on transient GitHub API errors when submitting formal PR reviews #159 and provides clear rationale for the changes. Human approval is always required for protected-path changes, regardless of review agent findings.

Low

  • [scope-creep] scripts/post-review.sh — The fallback applies labels for all actions (approve, comment, reject), but issue Post-review.sh should retry on transient GitHub API errors when submitting formal PR reviews #159 only specifically authorized fallback for the approve→ready-for-merge case. The extension to other actions is a reasonable engineering decision but exceeds the issue's explicit scope.

  • [missing-requirement] scripts/post-review.sh — Issue Post-review.sh should retry on transient GitHub API errors when submitting formal PR reviews #159 requirement (4) asks for HTTP status code and response body in error messages, but the retry loop only logs the exit code. This is partially limited by fullsend post-review not exposing HTTP-level details.

  • [code-duplication] scripts/post-review.sh — The fallback duplicates the outcome-label logic from the normal path. The comment acknowledges this ("keep in sync"), but a shared function could reduce maintenance burden and prevent the divergence noted in the [logic-error] finding above.

  • [minor-behavioral-difference] scripts/post-review.sh — The fallback removes all three outcome labels (including the one about to be re-added) before applying the new one, creating an unnecessary remove+add cycle on the PR timeline. The normal path skips removing the target label to avoid this.

  • [injection] scripts/post-review.sh — The ::warning:: command in the fallback interpolates ${ACTION} without sanitization for :: sequences or encoded newlines. The existing label-actions code (lines 264-269) explicitly sanitizes LA_ACTION for this exact pattern — the same defense-in-depth should apply here.

  • [missing-variable-documentation] docs/review.md — New environment variable POST_REVIEW_RETRY_DELAY is not documented in the Variables section alongside REVIEW_FINDING_SEVERITY_THRESHOLD.


Labels: PR modifies review pipeline post-review scripts

@fullsend-ai-review fullsend-ai-review Bot added requires-manual-review Review requires human judgment review-agent labels Jul 22, 2026
@rh-hemartin

Copy link
Copy Markdown
Member

/fs-fix

@fullsend-ai-coder

fullsend-ai-coder Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 9:52 AM UTC · Completed 9:59 AM UTC
Commit: 7ada4e0 · View workflow run →

- Document the assumption that fullsend post-review's exit code does not
  distinguish "comment posted, review failed" from "nothing posted",
  and note the HTTP status code limitation (findings 1, 5)
- Document intentional omission of gh pr close in the reject fallback
  as a conservative measure for uncertain state (finding 2)
- Skip removing the target outcome label in the fallback's stale-label
  loop to avoid a pointless unlabel/relabel cycle (finding 7)
- Sanitize ACTION for GHA workflow command output in the fallback
  warning message, matching the existing label-action pattern (finding 8)
- Document POST_REVIEW_RETRY_DELAY in docs/review.md Variables table
  (finding 9)

Addresses review feedback on #161
@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 1 (human-triggered)

Addressed 6 of 9 review findings with code changes: documented degraded-mode assumptions and HTTP status limitation, documented intentional omission of PR close in reject fallback, fixed stale-label removal to skip target label, sanitized ACTION in GHA warning output, and documented POST_REVIEW_RETRY_DELAY. Disagreed with 3 findings (protected-path is informational, scope-creep is a reasonable design choice, code-duplication refactoring is out of scope).

Fixed (6):

  1. degraded-mode fallback applies labels without verifying review comment was posted (scripts/post-review.sh): Documented the assumption explicitly: fullsend post-review does not emit distinct exit codes for 'comment posted, review failed' vs 'nothing posted', and HTTP status codes/response bodies are not exposed by the CLI (also addresses missing-requirement finding re: issue Post-review.sh should retry on transient GitHub API errors when submitting formal PR reviews #159 requirement 4).
  2. degraded-mode fallback for reject omits gh pr close (scripts/post-review.sh): Added comment documenting that closing is intentionally omitted in degraded mode as a conservative measure — without confirmation that the formal review was posted, closing the PR would be a destructive action based on uncertain state.
  3. HTTP status code and response body not in error messages (scripts/post-review.sh): Documented the limitation in the fallback comment block: fullsend post-review does not expose HTTP-level details, so only exit codes are available. Cross-referenced issue Post-review.sh should retry on transient GitHub API errors when submitting formal PR reviews #159 requirement 4.
  4. fallback removes all labels then re-adds target, creating unnecessary timeline noise (scripts/post-review.sh): Added _fallback_label computation before the stale-label removal loop and skip logic matching the normal outcome-label path, so the target label is not removed then immediately re-added.
  5. ACTION interpolated in ::warning:: without sanitization for GHA command injection (scripts/post-review.sh): Added sanitization of ACTION (strip newlines, carriage returns, and :: delimiters) before using it in the ::warning:: fallback message, matching the existing defense-in-depth pattern from label-action value sanitization.
  6. POST_REVIEW_RETRY_DELAY not documented in docs/review.md (docs/review.md): Added POST_REVIEW_RETRY_DELAY to the Variables table with description, default behavior (progressive 5s/15s backoff), and valid values.

Disagreed (3):

  1. PR modifies files under scripts/ (protected path): Informational finding, not a code fix. Human approval is required regardless of agent findings — acknowledged, no code change needed.
  2. fallback extends to all actions beyond issue Post-review.sh should retry on transient GitHub API errors when submitting formal PR reviews #159 explicit scope: The extension to comment and reject actions is a reasonable engineering decision. The fallback already exists and handles these cases correctly. Removing coverage would reduce robustness without clear benefit.
  3. fallback duplicates outcome-label logic — extract shared function: Per AGENTS.md simplicity-first and surgical-changes principles, extracting a shared function from shell code with global variable dependencies adds complexity without clear benefit in a ~500-line script. The 'keep in sync' comment is sufficient, and the label-skip divergence (finding 7) was fixed directly.

Tests: passed

Updated by fullsend fix agent

@rh-hemartin

Copy link
Copy Markdown
Member

/fs-review

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 22, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 10:06 AM UTC · Completed 10:20 AM UTC
Commit: 7ada4e0 · View workflow run →

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated 3-agent review pass (2 Claude-based agents + a Grok-based agent), focused on the retry loop's double-posting/idempotency risk, backoff correctness, and premature decisions, per the PR's own stated intent.

2 HIGH findings on the new retry loop: a missing idempotency guard against double-posting the formal review, and a set -euo pipefail interaction that lets an empty/non-numeric POST_REVIEW_RETRY_DELAY crash the script before the degraded-mode fallback runs. 3 MEDIUM findings: two premature-decision items (blanket transient-failure assumption; an overstated CLI-limitation claim tied to issue #159's requirements 2/4) and one test-coverage gap that let the crash bug ship undetected.

Prior review rounds' findings on this PR (label-fallback-without-verification, reject/close omission, label churn, GHA ::warning:: injection, POST_REVIEW_RETRY_DELAY doc gap, protected-path/scope-creep/code-duplication) are already fixed, documented as intentional, or declared out of scope, and are not repeated here.

Comment thread scripts/post-review.sh
Comment thread scripts/post-review.sh Outdated
Comment thread scripts/post-review.sh
Comment thread scripts/post-review.sh Outdated
Comment thread scripts/post-review-test.sh
@rh-hemartin

Copy link
Copy Markdown
Member

/fs-fix

@fullsend-ai-coder

fullsend-ai-coder Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 3:04 PM UTC · Completed 3:14 PM UTC
Commit: 7ada4e0 · View workflow run →

- Add idempotency guard: snapshot review ID before retry loop, check
  for newly-posted review before each retry to prevent double-posting
  if a "failed" attempt succeeded server-side (HIGH finding 1)
- Validate POST_REVIEW_RETRY_DELAY with regex (^[0-9]+$) instead of
  ${VAR+x} — empty/non-numeric values now fall through to the
  progressive default instead of crashing under set -e (HIGH finding 2)
- Capture fullsend stderr per attempt and include first line in
  ::warning:: messages for diagnostics (MEDIUM finding 4)
- Document retry-all-non-0/non-10 as deliberate trade-off since
  fullsend lacks distinct exit codes for failure classes (MEDIUM finding 3)
- Correct degraded-mode comment: the CLI does expose diagnostic output
  on stderr; the limitation is no distinct exit codes (MEDIUM finding 4)
- Add 3 test cases exercising default backoff (unset), empty-string,
  and non-numeric POST_REVIEW_RETRY_DELAY overrides (MEDIUM finding 5)

Addresses review feedback on #161
@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 2 (human-triggered)

Addressed all 5 review findings (2 HIGH, 3 MEDIUM). Added idempotency guard for the retry loop, fixed POST_REVIEW_RETRY_DELAY crash on empty/non-numeric values, captured stderr for diagnostics, documented retry-all as deliberate trade-off, corrected CLI limitation comment, and added 3 tests for default backoff and invalid delay values.

Fixed (5):

  1. Retry loop has no idempotency guard against double-posting the formal review (scripts/post-review.sh): Added pre-loop snapshot of the latest review ID via gh api, and an in-loop idempotency check before each retry. If a new review appeared despite the error exit code, the retry is skipped and POST_REVIEW_EXIT is set to 0 to avoid double-posting.
  2. Empty-string or non-numeric POST_REVIEW_RETRY_DELAY crashes the script before degraded-mode fallback (scripts/post-review.sh): Replaced ${POST_REVIEW_RETRY_DELAY+x} check with regex validation [[ "${POST_REVIEW_RETRY_DELAY:-}" =~ ^[0-9]+$ ]]. Empty, unset, or non-numeric values now fall through to the progressive default (5s/15s) instead of crashing sleep under set -e.
  3. Every non-zero/non-10 exit code is treated as transient without confirming fullsend's exit-code taxonomy (scripts/post-review.sh): Documented 'retry all non-zero/non-10 codes' as an explicit deliberate trade-off in the retry loop header comment, noting that fullsend does not expose distinct exit codes for transient vs. permanent failures and a permanent failure wastes at most two retry attempts (~20s).
  4. Comment states the CLI doesn't expose HTTP status/response body as settled fact; the CLI's own error formatting shows otherwise (scripts/post-review.sh): Captured stderr from fullsend post-review per attempt (2>tempfile) and included the first line in ::warning::/::error:: messages. Corrected the degraded-mode comment to accurately state the limitation is 'no distinct exit codes' rather than 'no diagnostic output available'.
  5. New retry tests all hardcode POST_REVIEW_RETRY_DELAY=0, so none exercise the production default backoff or an invalid override (scripts/post-review-test.sh): Added 3 standalone test cases: default-backoff-unset (leaves POST_REVIEW_RETRY_DELAY unset, asserts 'retrying in 5s'), invalid-delay-empty-string (exports empty string, asserts progressive default), and invalid-delay-non-numeric (exports 'abc', asserts progressive default). Also updated both gh mocks to handle the reviews API endpoint used by the new idempotency guard.

Tests: passed

Decision points
  • Chose lightweight review-ID comparison for idempotency guard over timestamp-based or user-identity-based approaches (alternatives: Check reviews by user identity and head SHA, Track review count before/after, Skip idempotency guard and only document the risk; rationale: Review ID comparison is the simplest reliable check: snapshot max review ID before the loop, compare after each failure. Falls back gracefully to 0 if the API call fails, preserving existing retry behavior.)

Updated by fullsend fix agent

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recheck pass on commit 84f3128 (previous pass reviewed c39f68b)

All 5 findings from the previous review pass are fixed in the current head and their threads have been resolved:

  • Idempotency guard added (snapshot + per-retry check against the reviews API)
  • POST_REVIEW_RETRY_DELAY empty/non-numeric values now validated via regex instead of crashing
  • Exit-code retry-taxonomy trade-off now documented explicitly
  • fullsend CLI stderr now captured and surfaced in warning/error messages; the comment's claim about CLI diagnostics corrected
  • Three new tests cover unset/empty/non-numeric POST_REVIEW_RETRY_DELAY

This pass dispatched 4 independent reviewers (2x Claude, Grok, Codex) against the current diff, focused on whether those fixes are actually sound and whether the fix commits introduced anything new. All 4 independently converged on a real gap in the idempotency guard itself (see inline finding below). Findings are posted inline where they land on diffed lines; one documentation finding falls outside this PR's diff hunks and is included here instead:

[MEDIUM] docs/review.md's Control-labels table (the ready-for-merge / requires-manual-review rows, not touched by this PR's diff) doesn't mention that either label can now be applied via the degraded-mode fallback without a confirmed posted review. Worth a follow-up doc note given the fallback-condition finding below — label presence no longer strictly implies a review was successfully posted.

Two single-agent findings below were independently verified against the fullsend CLI source and issue #159's text before inclusion. A third single-agent finding's supporting evidence (an internal retry-count claim) contained a factual error that's corrected in the finding text. Everything below MEDIUM severity was discarded per this pass's verification threshold.

Comment thread scripts/post-review.sh
_current_review_id=$(gh api "repos/${REPO_FULL_NAME}/pulls/${PR_NUMBER}/reviews" \
--jq 'map(.id) | max // 0' 2>/dev/null) || true
_current_review_id="${_current_review_id:-0}"
if [ "${_current_review_id}" -gt "${_last_review_id}" ]; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[HIGH] Idempotency guard has no author/commit filter and fails open on its own snapshot query

Finding: The new idempotency guard (_last_review_id snapshot at lines 343-345, comparison at lines 379-388) treats any increase in max(.id) across all PR reviews as proof that this run's own fullsend post-review call succeeded server-side — it never filters by posting identity or by commit_id matching the reviewed SHA. Two independent ways this produces a false "already posted" conclusion, both of which set POST_REVIEW_EXIT=0 and fall straight into the unconditional "Outcome labels" section: (1) if a human or another bot submits any review on the PR during the 5-20s backoff window, the guard concludes the failed attempt "actually succeeded"; (2) the initial snapshot swallows gh api failures via 2>/dev/null || true, silently defaulting _last_review_id to 0 — on a PR that already has prior reviews (e.g. from an earlier /fs-review redispatch cycle, which this pipeline explicitly supports), the very next per-retry check will see _current_review_id > 0 and falsely conclude success on the first failed attempt. In both cases the only trace is a low-visibility ::notice::, not a ::warning::/::error:: — the intended review is never posted, yet the workflow reports success and can still attach an outcome label.

Suggestion: Filter the reviews query to this run's own identity (cache gh api user --jq .login once, or resolve the bot login) and/or match on commit_id equal to the reviewed head SHA, e.g. --jq --arg sha "$HEAD_SHA" 'map(select(.commit_id == $sha)) | map(.id) | max // 0'. Treat a failed snapshot query as "unknown" rather than 0, since 0 makes the guard trivially satisfied by any pre-existing review.

Independently flagged by all 4 reviewers in this pass (Claude x2, Grok, Codex); one reproduced the false-positive empirically by simulating a permanent fullsend failure alongside an incrementing (but unrelated) reviews-API mock.

Comment thread scripts/post-review.sh
# codes do not differentiate failure modes.) Applying the label here is a
# best-effort measure — exit 1 still signals CI failure so a human can
# verify.
echo "Attempting degraded-mode label fallback..."

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[HIGH] Degraded-mode fallback applies the outcome label without confirming the review comment was actually posted

Finding: Issue #159's proposed change and validation criteria both explicitly condition the fallback on the comment having been posted ("if all retries are exhausted but the review comment was already posted successfully, attempt to apply the label"). The implementation drops that condition: the comment block above this line says outright "We cannot distinguish 'comment posted, review failed' from 'nothing posted'... Applying the label here is a best-effort measure" — and then applies it unconditionally whenever all 3 attempts exit non-zero/non-10. Tracing fullsend post-review's actual implementation confirms this matters: it posts the sticky comment first and returns immediately if that call itself fails, before ever attempting the formal review. So a failure mode common to all 3 attempts (e.g. an invalid/expired token, or the API being entirely unreachable) means nothing is posted to the PR — no comment, no review — yet the fallback still applies e.g. ready-for-merge for an approve verdict, purely from the sandboxed agent's proposed JSON action. The only signal anything went wrong is the workflow's exit code, which is easy to miss if downstream automation gates on label presence — the exact failure mode issue #159 was originally filed over.

Suggestion: Before applying any fallback label, verify a sticky comment actually landed — e.g. check the PR's issue comments for the <!-- fullsend:review-agent --> marker the CLI itself uses — and only proceed with the fallback if the marker is present, matching the issue's stated condition.

Flagged by 1 of 4 reviewers, who traced the actual call order and marker constant in the fullsend CLI source; independently confirmed here against the same source and against issue #159's text.

Comment thread scripts/post-review.sh

# Snapshot the latest review ID before the retry loop for idempotency.
# If a "failed" attempt actually posted a review, the retry detects it.
_last_review_id=$(gh api "repos/${REPO_FULL_NAME}/pulls/${PR_NUMBER}/reviews" \

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] Reviews-list API calls omit --paginate, silently capped at the first 30 (oldest) reviews

Finding: Both new gh api repos/.../pulls/.../reviews calls (lines 343, 381) fetch only GitHub's default first page — 30 items, oldest-first — unlike the other multi-page listing calls already in this file (labels --paginate, issues/.../comments --paginate). Once a PR accumulates more than 30 reviews over its lifetime (plausible for a heavily re-reviewed/re-dispatched PR under this same pipeline), max(.id) stays pinned to the same stale page-1 value on every check, and the -gt comparison can never fire — silently disabling the idempotency guard on exactly the busy, long-lived PRs most likely to need it.

Suggestion: Add --paginate to both calls, consistent with the existing convention in this file.

Flagged by 3 of 4 reviewers; one reproduced it directly (35 simulated reviews -> unpaginated query returns 30 regardless of the true max).

Comment thread scripts/post-review.sh
fi
echo "::warning::fullsend post-review attempt ${_pr_attempt}/${POST_REVIEW_MAX_ATTEMPTS} failed (exit ${POST_REVIEW_EXIT}${_pr_err_detail}) — retrying in ${_backoff}s"
sleep "${_backoff}"
else

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] Idempotency check is never re-run before the final "retries exhausted" branch

Finding: The idempotency re-check only runs inside if [ "${_pr_attempt}" -lt "${POST_REVIEW_MAX_ATTEMPTS}" ] (line 379) — i.e. between attempts 1->2 and 2->3. On the last attempt, a failure falls straight into this else branch ("all retries exhausted") with no re-query, then into the degraded-mode fallback and exit 1. If attempt 3 itself is a false negative — the server processed the write but the CLI reported failure, the exact scenario this whole PR exists to handle — the script has no way to detect it at the one moment it matters most, and reports a failure (re-applying a label via the fallback) for a run that may have actually succeeded.

Suggestion: Run the same review-ID comparison once more immediately after the final attempt fails, before falling into the elif [ "${POST_REVIEW_EXIT}" -ne 0 ] branch, so a genuinely successful last attempt is still reported as success.

Flagged by 3 of 4 reviewers.

#!/usr/bin/env bash

# gh api repos/.../pulls/.../reviews (idempotency guard)
if [[ "\$1" == "api" ]] && [[ "\$2" == *"/reviews"* ]] && [[ "\$*" == *"--jq"* ]]; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] premature-decision — No test exercises the idempotency guard's actual trigger path

Finding: This mock's reviews-API handler (and its non-retry counterpart earlier in the file) unconditionally echo "0"s regardless of call count. Across all 7 retry tests plus the 3 backoff/validation tests, _current_review_id can therefore never exceed _last_review_id — the actual anti-double-post branch this PR adds in post-review.sh (POST_REVIEW_EXIT=0; break on the idempotency-guard hit) has zero test coverage. This is the same untested-safety-mechanism pattern that let the original crash bug ship unnoticed last round: the guard added specifically to close a HIGH-severity risk is itself shipped with no test proving it fires.

Suggestion: Add a mock reviews handler that returns an incrementing max-id on a later invocation (mirroring the existing MOCK_FULLSEND_COUNTER pattern), and a test asserting the "idempotency guard" notice appears and no further fullsend call happens after it fires.

Flagged by all 4 reviewers.

Comment thread scripts/post-review.sh
# "Error: github api: <status> <message>" on API failures).
_pr_err_detail=""
if [ -s "${_pr_stderr}" ]; then
_pr_err_detail=" — $(head -1 "${_pr_stderr}")"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] Raw fullsend stderr is interpolated into GHA ::warning:: annotations without the sanitization used elsewhere in this file

Finding: _pr_err_detail is built from unsanitized fullsend stderr and embedded directly into the retry ::warning:: messages a few lines below. Elsewhere in this same file, external/derived strings destined for GHA workflow-command output are explicitly stripped of \n, \r, and :: before use — see LA_ACTION/LA_LABEL earlier in the file, and _safe_action (added by this very PR for a sibling message in the same fallback block, just further down). Since GitHub's validation-error bodies can echo back parts of the request that triggered them, a 422 whose message reflects PR-controlled input could plausibly propagate an unsanitized :: sequence into a workflow-command annotation.

Suggestion: Apply the same \n/\r/::-stripping already used for _safe_action to _pr_err_detail before interpolating it into the ::warning:: messages.

Flagged by 1 of 4 reviewers; verified here against this file's own established sanitization pattern.

Comment thread scripts/post-review.sh
# NOTE: fullsend does not expose distinct exit codes for transient vs.
# permanent failures (e.g. invalid token, malformed result file). As a
# deliberate trade-off, all non-zero/non-10 codes are retried — a permanent
# failure wastes at most two retry attempts (~20s) before falling through to

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] premature-decision — the "~20s worst case" comment doesn't account for fullsend's own internal HTTP retry/backoff

Finding: This comment states a permanent failure "wastes at most two retry attempts (~20s)." That holds only for failure classes the fullsend CLI doesn't already retry internally. fullsend's own GitHub client retries 429/5xx/secondary-rate-limited-403 responses internally (maxRetries = 3 in its GitHub client — not 5, as initially reported to me and corrected after checking the source directly), and for secondary rate limits specifically uses a 60s-plus-jitter base backoff — so a single fullsend post-review invocation can itself take roughly two minutes to give up before ever returning control to this script. Stacking this shell-level retry loop (3 attempts) on top means the true worst case under sustained rate-limiting is on the order of several minutes, not "~20s" — though 422 (this PR's specific motivating failure) is not retried internally by the Go client, so the common case is unaffected.

Suggestion: Scope the "~20s" claim to failure classes the CLI doesn't already retry (e.g. 422, auth errors), or drop the specific number and note that total worst-case latency depends on fullsend's own internal retry behavior for rate-limit-class failures.

Flagged by 1 of 4 reviewers, who traced the retry constants in the fullsend CLI source; re-verified directly against that source here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

requires-manual-review Review requires human judgment review-agent

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Post-review.sh should retry on transient GitHub API errors when submitting formal PR reviews

2 participants